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Introduction 



R eform has been part of the experience of educators for the extent of their careers. The present 
emphasis on aligned instruction, community engagement, excellence in teaching, turning 
around low-performing schools, excellence in leadership, and a positive and safe school 
environment is not new. Educators have watched various reform strategies, including “con- 
tinuous school improvement,” “effective schools,” “whole school reform,” and “comprehensive school 
reform,” come and go. Today, these and other models are subsumed under the promise “to turn around 
low-performing schools.” This interpretation promises a far deeper, more extensive, and more sustained 
change effort than reforms of the past. Reform, no matter the performance level of the school, needs to be 
sustained to have an effect on student achievement. Evaluation can and should be a tool for sustaining 
school reform efforts, as well as measuring progress and impact. 

The use of data, which is a central component of the ARRA reform strategies, needs to include data 
for strategic decision making at the school level. Although evaluation must be solidly based on data, 
evaluation should go beyond data to examine the larger context of policy and practice, which either 
fosters success or sets the course for failure. An effective evaluation does not just assess the outcomes 
and impact of new strategies; it guides the process of implementing and refining reform programs. 

We come to this point in the history of education reform better equipped with research-based practices 
and with more access to them than ever before, through such resources as the What Works Clearinghouse 
Practice Guides (http://ies.ed.gov/ncee/wwc/publications/practiceguides/) and the U.S. Department of 
Education Doing What Works website (http://dww.ed.gov). There certainly will be new directions and 
innovations in education in the future, but that does not mean that current methods for creating better 
schools will have been proven wrong. Rather, it means that educators will have acquired new knowledge 
about how to make schools successful. More important, it means that researchers and educators have not 
tired of trying to make the nations schools the very best in the world. 

Although it might seem that for the foreseeable future, most of the attention and available resources 
will be deployed at the state level or for those schools identified as struggling or persistently lowest- 
achieving, at another level, Americas place in the global economy is at stake. To assure success in the 
global arena, we must be able to respond to a rapidly changing environment that demands new knowl- 
edge, new skills, and flexibility to meet emerging challenges. Every local school plays a role in assuring 
these new global challenges are met at a national level. 
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The American Recovery and Reinvestment Act (ARRA) of 2009 sets out nationwide goals for the 
improvement of American education: increasing student achievement, narrowing achievement gaps, and 
increasing high school graduation and college enrollment. It is built around four areas of reform: (1) 
internationally benchmarked college and career readiness standards; (2) improvement in the collection 
and use of data to inform decision making; (3) increase in teacher and leader effectiveness and improve- 
ment in distribution; and (4) support for struggling schools, with a focus on turning them around. 

Yet, the present ARRA school reform attempts to move beyond previous efforts by bringing together 
much of what has been learned about creating better schools. ARRA assumes the need for ambitious 
reform measures; provides additional, practical fiscal resources; and continues to hold the education 
system accountable for realizing the improved performance of all students. 

Ambitious policy reforms — including comprehensive school reform, which went beyond piecemeal 
improvement efforts by integrating strategies — have often fallen short of encouraging substantial 
increases in student outcomes. As educators David Cohen, Susan Moffitt, and Simona Goldin (2007) 
point out, “Policies aim to solve problems, yet the key problem solvers are those who have the problem. 
Governments devise instruments to encourage implementation, but they help only if used well by those 
with the problem” (p. 515). This guide seeks to encourage local educators to use the reforms well 
through collection and interpretation of data within a systematic evaluation plan. 

Purpose of This Guide 

This guide helps districts and schools, whether or not they have been designated as persistently 
low-achieving, to evaluate their reform efforts — both to meet requirements for external accountability 
and to meet their own needs for useful evaluative information. The information provided is applicable 
to any local jurisdiction that wants to know how well its turnaround initiatives are working and what 
additional changes are needed. 

We assume that your district or school has decided to implement a school reform initiative and has 
identified a model or a set of strategies to begin the process. We also assume that a plan is in place, funding 
is available, and that you are now facing challenges related to data collection and evaluation. Almost 
certainly, your new program will add a layer of data collection responsibility to an already overburdened 
staff. It is tempting in this situation to simply meet the external data requirements and stop there. However, 
conducting an ongoing evaluation allows you to better understand what is working and what isn’t, to make 
changes, and to report progress as your reform effort unfolds. 

Generating the information needed is completely under your control. Therefore, the evaluation 
challenges you face are a powerful opportunity for informed decision making at the local level. This 
guide will assist you through the process of program evaluation so that you will be better able to 

• conduct a focused, practical school-reform evaluation; 

• develop a detailed evaluation design; 
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• conduct an evaluation of school-reform implementation and impact; 

• report key findings in formats useful to key audiences; and 

• encourage use of evaluation findings for program improvement. 

The purpose of this guide is to provide practical information, tips, and tools, which collectively 
serve as a resource for novice evaluators as well as a reference for expert evaluators. 

Using This Guide 

Evaluation is never a “one size fits all” proposition. The specific details of each locally designed 
evaluation of school reform varies, depending on the approach crafted and the goals and benchmarks 
established to assess progress. This guide organizes the array of evaluation activities into a five-stage 
evaluation process that is consistent in timing and purpose: 

1. Planning for the evaluation 

2. Designing the evaluation 

3. Conducting the evaluation 

4. Reporting the findings 

5. Encouraging use of the findings 

Your evaluation team may photocopy and use Worksheets 1-8 (see Appendix A) during the evaluation 
process. Throughout this guide, Exhibits 1-8 provide examples of how the completed worksheets might 
look. The exhibits are not exemplars of what a particular district or school should try to create; they are 
illustrations of a range of situations, responses, and data options that could emerge. 

If you are an experienced evaluator who is primarily interested in utilizing the worksheets, you may 
quickly reference the corresponding exhibits by scanning for the headings in brouMs italics (e.g., “Usuuj 
Worksheet 1 to AlujKs Rc^turmtentt’ ) or finding these headings in the table of contents. In addition, look 
for the shaded boxes of “evaluation tips” that appear at the end of the text for each stage. 

Each main section in this guide focuses on a particular stage of evaluation. The stages are presented in the 
sequence in which they are typically conducted. If you are just beginning an evaluation, it may be most 
beneficial to read through the chapters in sequence. If you have already begun an evaluation, you may want to 
begin with the stage that best matches the evaluation stage you are in, returning to earlier chapters as needed. 
Regardless of the stage of evaluation that your school or district is involved in, as the evaluation progresses, you 
may revisit chapters to help as you modify your original design, data collection methods, or analysis plan. 
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School Reform Under ARRA 

With the passage of ARRA, funding for school improvement greatly increased. As the programs within ARRA 
have unfolded, school improvement has consistently been defined and applied in each of them. Thus, while we 
draw from the guidance for the School Improvement Grants (SIGs), the recommendations we make can be 
applied across all ARRA reform areas. 

Requirements for awarding and monitoring SIGs were published in the Federal Register (School Improvement 
Grants, December 10, 2009). Although not all schools and districts have received or will receive a SIG, the 
guidance contained in the Register can assist all districts in knowing whether a reform intervention results in 
improved student achievement. As the Federal Register (School Improvement Grants, August 26, 2009) 
reminds, 

Local educators need the data on an ongoing basis to evaluate the extent to which effective reform 
strategies are being implemented, to monitor the impact of changes, to track progress against their own 
goals, and to identify areas where, during implementation, assistance or adjustments are needed, (p. 

43107) 

For purposes of assessing the SIG program, the federal government has proposed data collection in three 
categories: (1) the intervention being implemented, (2) leading indicators (minutes per school 
year and teacher attendance rate), and (3) student achievement outcomes (average scale scores on state assess- 
ments for the “all students” group, for each achievement quartile, and for each subgroup and number of 
students enrolled in advanced courses). 

Background Assumptions 

Your school or district may not have received a SIG, nor been identified as a persistently low-achieving school. 
Nonetheless, the four core areas of reform outlined in ARRA reflect the best thinking for reform at this time. All 
schools are well advised to plan their reforms in this context. The models of intervention for states to implement in 
turning around struggling schools include major structural changes. These changes even include replacement of staff; 
yet, reconstituted staff still need to implement the reforms. Thus, four areas are relevant to all schools: (1) developing 
and increasing teacher and school leader effectiveness, (2) comprehensive instructional reform strategies, (3) increas- 
ing learning time and creating community-oriented schools, and (4) providing operating flexibility and sustained 
support. 

The evaluation of a reform initiative is built on the plans, strategies, and particulars of the interventions that the 
school or district has in place. Before you begin planning the evaluation, obtain copies of all reports or documents 
used in choosing or shaping the reform interventions. 


Sustaining 
Coherent 
Reform 

An Evaluation 
Guide for 
Districts and 
Schools 

and about the impact and outcomes realized. Because many outcomes take considerable time to emerge, it 
is important to maintain a continuous flow of evaluative information, making corrections and adjustments 
along the way to increase the likelihood that outcomes will be realized. Four suggestions follow. 

Examine the process and the impact. An effective program evaluation documents the interme- 
diate and long-term outcomes of the reform effort. It also examines the adequacy of implementation 
processes in view of the outcomes observed. In short, a truly useful evaluation provides feedback to decision 
makers and participants about needed modifications and improvements. Evaluation, then, needs to be an 
integral and ongoing part of the reform program and to be embedded in program operations — rather than 
tacked on as an additional administrative task to meet someone elses requirements. Thus, evaluation 
planning is best begun in the early stages of the program, during the start-up phase. 

Create a flexible design. Just as you will adjust the reform program when necessary, so too you 
will modify the evaluation design as needed. Your design, then, must be flexible enough to allow for 
modifications. A flexible design empowers you to respond to feedback about unanticipated problem 
areas. If a program element is not working well, you must be able to adjust your evaluation plan in 
order to obtain the needed evaluative information. For example, if an instructional reform is not being 
embraced and implemented in classrooms as expected, you could revise the evaluation to include a 
focus group of teachers to assess the nature of the problem and to surface their concerns. 

Keep the design realistic. Keep the evaluation effort as comprehensive and rigorous as possible, 
yet realistic in scope and demand, given available resources. There are many aspects of a complex 
program that can be evaluated. Your job is to identify the most important questions and give them 
priority. Data collection can be expensive, but time and money are almost always limited, so direct 
resources to learning about the most critical program components and issues. 

One strategy for using scarce resources wisely is to be clear about why particular information is being 
collected. It is also useful to know in advance how data sets will be used to answer specific questions. If 
there is no clear reason to collect a data set (other than that it is available or relatively easy to obtain), 
eliminate it from the evaluation design. A carefully chosen evaluation consultant, brought in at the design 
stage, can help identify cost-effective ways to focus your evaluation so that you can maintain acceptable 
standards while maximizing available resources, (see Stage 1, Step 4 for more about selecting consultants.) 


What You Should Know About Evaluation 

Often, the best approach to carrying out educational evaluations is an eclectic one, combining what 
appears to be best from various methods rather than adhering to a particular evaluation model. The 
visibility of an evaluation can be increased by considering it an integral component of the overall reform 
effort. The evaluation should provide information to all stakeholders about progress toward stated goals 
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Establish an evaluation team. An evaluation team is essential to the successful design and 
ongoing maintenance of an evaluation effort. Establish a small, core team of staff members, and plan 
opportunities for frequent discussion and coordination throughout the building. Also give each team 
member release time to focus on these new duties. 

Designate one person as the point of contact for evaluation-related issues and questions; this team 
leader is responsible for convening the evaluation team and ensuring that others are included in all 
stages of the evaluation effort as appropriate. The team as a whole is responsible for acquiring outside 
expertise and assistance as needed. A larger team that includes community members can serve as an 
advisory group for the core team. 


Stage 



Planning for the Evaluation 


D iscussing the details of an evaluation 
design (e.g., evaluation questions, 
measures, methods, analysis) is often 
where evaluators begin the evaluation 
process. Resist this temptation; take time to think 
and plan first! Considerable time and effort can be 
saved by completing four planning steps: 

Step I. Define the purpose(s) of the evaluation. 
Step 2. Identify program evaluation 
requirements. 

Step 3. Understand the evaluation context. 

Step 4. Consider using a consultant. 

Step I. Define the Purpose(s) 
of the Evaluation 

Be clear about what you expect your evaluation 
to accomplish. At a minimum, an effective 
evaluation provides the data needed to meet the 
reporting requirements of the reform initiative. 
Equally important, however, is the evaluation 
of outcomes — and the evaluation processes 
used to attain them. Your evaluation should 
examine progress, assess the accomplishment 
of intermediate objectives, and provide findings 
in a timely manner so you can make needed 
programmatic adjustments as you move forward. 
Record your purpose on Worksheet 1 . 


Step 2 . Identify Program 
Evaluation Requirements 

Among your first tasks is the identification of all 
evaluation requirements to be met. Requirements 
for designing your school reform evaluation are 
plentiful and come from several sources, including 

• district and/or building constituents, 

• state departments of education, 

• other funder requirements, and 

• the federal government. 

Federal and state evaluation requirements are 
important, but if you are at the school level, there 
may be other requirements from the reform 
program chosen or your local school board may 
have set requirements. 

Federal guidelines. A review of the federal 
rules and regulations for SIGs found in the Federal 
Register (School Improvement Grants, December 
10, 2009) reveals the following requirements: 

• An LEA must use rigorous, transparent, and 
equitable evaluation systems for teachers and 
principals that take into account data on 
student growth and other factors, such as 
observation-based assessments of performance 
and collections of professional practice, and 
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Four Types of School 
Improvement Models 

Turnaround: Replace principal and rehire no more 
than 50% of the staff; adopt new governance; begin a 
new or revised instructional program; implement 
interventions that take into account the recruitment, 
placement, and development of staff; execute schedules 
that increase time for students and staff. 

Restart: Close the school and restart under the 
management of a charter-school operator, charter 
management organization, or EMO; admit (within 
school’s grades) former students who wish to attend. 

School Closure: Close the school and enroll the 
former students in other higher performing schools 
in the LEA. 

Transformation: Implement comprehensive 
instructional programs using achievement data; extend 
learning time and create community-oriented schools; 
provide operating flexibility and intensive support. 


that are designed with teacher and principal 
involvement (p. 65620). 

• The continuous use of student data to 
inform and differentiate instruction must 
be promoted to meet the academic needs 
of individual students (p. 65635). 

• An LEA must make progress on the leading 
indicators (minutes per school year and teacher 
attendance rate) and establish annual goals for 
student achievement on the state s assessments 
in both reading/language arts and mathematics 
(p. 65641). 

• An LEA may also conduct periodic reviews 
to ensure that the curriculum is being 
implemented with fidelity, is having the 
intended impact on student achievement, 
and is modified if ineffective (p. 65651). 

In addition, the Federal Register (School 
Improvement Grants, 2009) encourages LEAs 
to “collect and use any data above and beyond 
these requirements that they believe will assist in 
the effective implementation of the four school 
intervention models” (p. 65648). 

Appendix B includes a list of the data which 
are collected by Facts: some are new, collected 
under the Stimulus Fund data requirements for 
state education agencies (SEAs). Other data are 
to be collected by any LEA that receives a SIG. 

State and district requirements. State 
department of education solicitations for 
proposals are another source of specified 
evaluation guidelines to meet. Also, your state 
probably has required the districts and buildings 
it funds to address evaluation in a specific way. 
Although you may feel that your proposal has 
responded completely to state requirements, 
it is worth another look. Continued funding may 


depend on compliance with evaluation and other 
accountability requirements. 

Reform model requirements. This 
guide is primarily responsive to the school- 
improvement funding authorized under ARRA, 
but it also is useful to any school undertaking an 
improvement effort. Under the ARRA funding, 
four types of school improvement models (see 
sidebar, “Four Types of School Improvement 
Models”) are defined: turnaround, restart, school 
closure, and transformation. The turnaround and 
restart models focus on major organizational 
change. In both cases, after the requirements for 
staff changes are met, reforms must be planned and 
evaluated. For the school closure model, the district 
may choose to track the placement and subsequent 
achievement of the relocated students. In regard 
to the transformation and turnaround models, 


i: Example of Completed Worksheet l Worksheet 1: Alignment of Evaluation Requirements 
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you can expect major changes in four areas: 
developing and increasing teacher and school leader 
effectiveness, comprehensive instructional reform 
strategies, increasing learning time and creating 
community-oriented schools, and providing 
operating flexibility and sustained support. 

Using Worksheet 1 to Aligns ke^utreuimti 

Use Worksheet Is Alignment of Evaluation 
Requirements to organize and align the various 
evaluation requirements that your reform must 
meet for federal, state, and local agencies. Exhibit 1 
illuminates the sets of evaluation requirements that 
overlap. Recognizing this redundancy at the outset 
can improve the efficiency and effectiveness of your 
evaluation design. Exhibit 1 does not include the 
required student-outcome data already being 
submitted through ED Facts (see Appendix B). 

Step 3. Understand the 
Evaluation Context 

It is important to understand the local context 
in which your program and the evaluation are 
to be implemented. Contextual factors may 
affect your choice of evaluation design and 
data-collection strategies, as well as how key 
stakeholders use the results. Thus, the objective 
of this step is to “get the lay of the land” and 
set the boundaries of your evaluation. 


Using Worksheet 2 to Organize 
Planjting Information; 


To organize your planning, consider four 
areas: ( 1 ) local needs, ( 2 ) program elements, 

(3) stakeholder expectations, and (4) evaluation 


resources and expertise. Use Worksheet 2: 
Evaluation Planning Questions to respond to 
questions related to these four areas and to 
organize the information you develop as a result. 
Exhibit 2 offers possible responses to each 
question to stimulate thinking and illustrate the 
types of responses you can expect to encounter. 
Worksheet 2 will help you organize the planning 
information you collect in interviews and 
document review. 

Local needs. To gain a clear understanding 
of needs or conditions that can be improved, 
examine school reports but also obtain the views 
of community members and school staff. Attempt 
to answer the following question: 

What local education needs or conditions 

underlie the school's current need for reform? 

Program elements. When describing the 
intent of your reform, you may include a 
synopsis of classroom activities, support services, 
and model components. Your application (or 
LEA application) may be the best source of 
information to describe the intent of the reform. 
Upon review, you may find that descriptions in 
the application are not clear or not sufficient to 
meet planning needs; therefore, interpretation 
may be necessary. You may also need to consider 
additions or changes that have been made to the 
plan since the application was written. Attempt 
to answer the following two questions: 

What are the goals and objectives of the reform? 

What are the major elements of the reform that 

are pertinent to the evaluation? 

Stakeholder expectations. Stakeholders 
are those people who will be involved in the reform 
or who are concerned with its success. These 
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exhibit 2 : Example of Completed Worksheet 2 Worksheet 2: Evaluation Planning Questions 
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individuals or groups will have much to say about 
your efforts and their results. By identifying them 
early in the evaluation process, you can involve 
them in the evaluation design process as much as 
possible. One way to begin identifying stakeholders 
is to learn who was involved in planning and 
writing the SIG or other application. Identifying 
stakeholders and their expectations is only part of 
the task. You also will want to discern what will 
convince them that the reform is successful. 
Attempt to answer the following questions: 

Who is interested in the reform and what do 

they expect from it? 


What will be accepted as credible evidence of 
progress and impact by each individual or group? 


Resources and expertise. Before design- 


ing your evaluation, determine the extent of 
financial and personnel resources available to the 
evaluation effort. Think beyond the design phase 
to the need for ongoing data collection, analysis, 
and reporting. Funds for evaluation may have been 
earmarked in your district s application or not 
considered at all. 

What budget can be allocated for evaluation? 
You may need to include several categories, 
such as salaries, material development, copying, 
postage, printing, and consultants. Estimating 
costs before a detailed design is in place is always 
difficult; however, experience shows that a 
reasonable evaluation budget can be estimated 
at 10 percent of the total reform cost. Costs 
will be somewhat less if you can pool resources 


Tips for Planning the Evaluation 

■ When considering the purposes of your evaluation, be sure to include monitoring and 
modifying the efforts as well as demonstrating outcomes. 

■ Start thinking now about how and by whom evaluation findings will be used. 

■ To get a feel for local conditions, review school reports and local newspaper articles; and/or 
interview community members, parents, administrators, and teachers. 

■ To learn about the purposes of your reform, examine the application and talk with those 
who wrote it. 

■ Find out who has an interest in the outcome of the reform and what they expect. If differences 
exist between reform goals and stakeholder expectations, stop planning and resolve the 
discrepancies immediately. 

■ Identify staff time, expertise, and resources allocated to complete the evaluation. 

■ If resources are insufficient to conduct an adequate evaluation, consider working with nearby 
schools, districts, colleges, or education service agencies that may be able to contribute staff 
time or material assistance as part of their required programming. 

■ Identify key time frames for evaluation information, such as federal and state reporting cycles, 
community forums, and legislative sessions. 
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and develop designs and materials that multiple 
schools or districts can use. It is easier and 
advisable to estimate or reassess the cost of 
evaluation after you have completed your design. 

Finally talk with those you expect to be 
members of the evaluation team. Assess the 
evaluation skills and experience of the team 
in order to decide whether or not to use an 
outside consultant (see Step 4 for more about 
consultants). In reviewing the application and 
discussing evaluation expectations, attempt to 
answer the following questions: 

What resources and technical expertise are 
available to support the evaluation ? 

What additional resources and expertise 
are needed ? 

Step 4. Consider Using 
a Consultant 

An outside consultant can assist at virtually 
every stage of your evaluation. During the 
early phases, the consultant can help frame the 
evaluation questions and identify or modify 
existing data-collection instruments. Later in 
the process, the consultant can train staff in 
data-collection methods, develop additional 
instruments, and create systems for organizing 
and monitoring the evaluation process. Finally, 


the consultant can assist with analyzing and 
interpreting results and help design the 
multiple reports for different stakeholder 
groups. Consider the tasks to assign to the 
consultant and the phases of the evaluation 
in which you will need assistance. 

There is no licensing or credentialing system 
for evaluators or consultants. Therefore, 
gather as much information about prospective 
consultants as possible. Obtain a resume or 
vitae and get references directly from those 
who have worked with them. Ask the 
consultant for the names of former clients 
whose projects are similar to yours. Have the 
consultant submit a two- or three-page brief 
that describes how he or she is particularly 
qualified to assist you with your evaluation, 
what approach or method will be used, and 
what key issues need to be discussed and 
resolved to complete a successful evaluation. 
Before making a final selection, consider 
the adequacy and appropriateness of each 
prospective consultant in view of his or her 

• formal preparation in evaluation, 

• relevant experience, 

• philosophy or orientation to evaluation, 
and 

• personal style and characteristics. 
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Stage 2 



Designing the Evaluation 


n Stage 2, you will design an evaluation of 
program context, implementation, and outcomes. 
Stage 2 recommendations are based on the 
assumption that you have clarified the context 
and purpose of your program and have established 
an implementation strategy. Typically, the 
implementation strategy recognizes not only 
the goals or intended long-term outcomes but also 
the immediate outcomes, such as increased teacher 
knowledge. You must know the purpose, context, 
and implementation strategy before you can select 
appropriate measures of intermediate outcomes 
and impact (to assess progress) and appropriate 
measures of long-term outcomes and impact (to 
assess the value of time and money invested). 

There are many steps involved in designing an 
evaluation, and most of them are interrelated. 

What you learned in Stage 1 and documented with 
Worksheets 1 and 2 serves as a starting point for 
developing an evaluation design. Use the information 
to check the adequacy of your final evaluation 
design. Four steps are completed in the design stage: 

Step I. Link the evaluation to the program. 
Step 2. Identify the evaluation questions. 

Step 3. Choose data sources, variables, 
and measures. 

Step 4. Move from planning to action. 


Step I. Link the Evaluation 
to the Program 

In a comprehensive reform initiative, the program 
components are expected to work together toward 
the ultimate goal of improved student achievement. 
To make connections among components, use 
Worksheet 3: Comprehensive School Reform 
Program Logic Model to develop a program logic 
model that specifies the assumptions that link 
program components. The program logic model 
usually begins on the left side of the page with the 
context of the reform, and then it lists the persons 
or conditions that the program seeks to change. 
Next, it identifies the activities that will cause 
change. Finally, it indicates the changed realities 
the program intends to accomplish. 

Review the logic of your program and identify 
gaps and inconsistencies. Will the activities 
logically result in the changes? Remember that 
the outcomes are only one focus of the evaluation; 
you will also focus on the implementation of the 
program and the school or district context. For 
each of these areas of focus, develop evaluation 
questions to frame the design (see Step 2 in this 
section for more about developing these questions). 

The logic model clarifies underlying assumptions 
of a program and specifies intermediate outcomes. 
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After you have developed your program logic 
model, you will be able to develop the program 
evaluation questions. Figure 1 illustrates 
how the program logic process leads to the 
identification of evaluation questions about 
context, implementation, and outcomes. 


Usuuj Worksheets to Create a. 
Program Lojlc Model 


Exhibit 3 shows how Worksheet 3: 
Comprehensive School Reform Program 
Logic Model was used to develop a logic model 
for a program to guide the development of the 
evaluation design and evaluation questions. 

In order to complete Worksheet 3, you will 
need your application, materials about 
implementation from the reform planners, 
and a completed Worksheet 2. The first step 


in completing Worksheet 3 is to list the local 
conditions, issues, or concerns that may 
influence the success of your reform effort 
(see the Contextual Issues column). 

The next step is to use the Program Elements 
column to list the key aspects of the program. It 
is not necessary to link specific program elements 
to specific issues listed in the Contextual Issues 
column. The program elements should not be 
specific tasks (such as “schedule the professional 
development sessions”). Rather, they should be 
broad; for example, “professional development” 
or “curriculum and assessment alignment.” Entries 
in the Assumptions and Intermediate Outcomes 
columns should be explicit statements of the 
underlying logic about how program elements 
lead to goals and outcomes. List the assumptions 
and intermediate outcomes underlying each 
program element in the boxes adjacent to each 



figure i: Identification of evaluation questions 
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3: Example of Completed Worksheet 3 Worksheet 3: Comprehensive School Reform Program Logic Model 
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element. You may want to write them as if/ then 
statements (as shown in Exhibit 3). For example, 
if professional development is a program element, 
your assumption might be “If teachers receive 
training in classroom management, then students 
will be on task more often.” 

For the program aspects and underlying 
assumptions you have specified, identify one or 
more intermediate outcomes associated with 
each, as shown in Exhibit 3. These outcomes are 
the “then” part of the if/ then statement. If you 
need to establish more frequent benchmarks, then 
generate several if/ then statements that create a 
sequence of events leading toward improved 
student achievement. Keep in mind, however, 
that you must collect data to determine if each 
outcome is being realized; therefore, it is best to 
limit the number of benchmarks to a carefully 
crafted set. Remember, all of the outcomes should 
move your school or district toward the ultimate 
goal of improved student achievement. 

Review your logic model one last time. Does 
your planned program make logical sense? If 
the intermediate outcomes are realized, will you 


accomplish your goals for improved student 
achievement? Are the assumptions underlying 
if/ then statements research based or lacking 
support? Are these the right set of activities to 
accomplish the intermediate goals? Are there 
gaps in the comprehensiveness of the model? 
Remember, there is little point in conducting 
an evaluation if the program is not well designed 
to accomplish its goals. 

Step 2. Identify the 
Evaluation Questions 

Usuuj Worksheet 4 to StmuMArtze 
EimIua£LoKs Questions 

Evaluation questions provide the framework 
of the evaluation design. Exhibit 4 includes 
examples of three types of evaluation 
questions — context, implementation, and 
outcomes — to be recorded on Worksheet 4: 
Summary of Evaluation Questions: 

Context questions address the factors that may 

influence your program and concerns related 


Tips for Designing the Evaluation 

■ Focus initial evaluation efforts on implementation questions.The findings obtained can be used 
to improve program operations. 

■ Plan for and schedule the collection of baseline data early in the project. 

■ Plan to use secondary data (i.e., existing data) whenever possible and integrate evaluation data 
collection into established site-specific procedures as much as possible. 

■ Coordinate the data collection and reporting requirements of your reform initiative with the 
requirements of other programs already in place. 

■ Look for opportunities to use the same measures to address multiple evaluation questions. 
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■: Example of Completed Worksheet 4 Worksheet 4: Summary of Evaluation Questions 
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to the unique characteristics of your school 
(e.g., changes in population or changes in the 
economic make-up of the district). 

Implementation questions are more practical and 
action-oriented. The overarching implementation 
question is, “Did you do what you said you 
would do?” 

Outcome questions focus on what has resulted 
from the program. “Have the intended goals or 
objectives been realized?” or “What impact has 
the program had?” 

Draft a list of questions for each of the three 
question types. Then revisit completed Worksheets 
1 and 2 to be certain you have included all the 
questions to address. 


Step 3. Choose Data Sources, 
Variables, and Measures 

One of the most important aspects of evaluation 
design is determining the best sources of information 
for answering the evaluation questions (see Figure 
2). Documents, people, and events all are likely 
sources. The best sources are those that are closest 
to the issue or topic you are evaluating, but using 
them may not be possible or cost effective. What 
you want to learn from each data source is defined 
by the variables and measures. 


Using Worksheet 5 to Creation 
Emluation Design Matrix 


The evaluation design matrix is one of the 
primary components of the evaluation design. 



figure 2: Relationship among data sources, types of information, and evaluation questions 
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exhibits: Example of Completed Worksheet 5 Worksheet 5: Evaluation Design Matrix 



Sustaining Coherent Reform An Evaluation Guide for Districts and Schools 


21 


Exhibit 5 provides an example of Worksheet 5: 
Evaluation Design Matrix. Begin Worksheet 5 by 
transferring the questions from Worksheet 4 to the 
first column. Then consider the best sources of 
data for each of the evaluation questions. As shown 
in the examples in Exhibit 5, sources of data may 
include existing records and documents (e.g., 
student test scores, equipment inventory), people 
(e.g., teachers, students, parents), and events (e.g., 
meetings, classroom teaching sessions). 

For each question and data source, identify the 
relevant variables and measures. The variables 
operationalize, or make specific, the evaluation 
questions. For example, for the context question 
“Are all students benefiting from the reform?” 

(see Exhibit 5), the variables are student attitudes 
and performance of students by race and gender 

As shown in Exhibit 5, measures are the 
specific means you will use to assign values 
to the variables. Using multiple measures of 
student performance is good practice. In 
addition to state assessments of student 
performance, review local- or school-developed 
assessments if they are available. These may be 
better aligned with your local curricula. Include 
additional measures of student and school 
performance in your evaluation to obtain a 
broad view of the value of your reform. Choose 
measures that reasonably can be related to 
program goals. Examples include attendance; 
grade promotion; graduation, suspension, and 
expulsion rates; course-taking patterns; and 
parent time in classrooms. When you have 
completed the Variables and Measures columns, 
you will know exactly what information you 
will use to answer the evaluation questions. 


Step 4 . Move from Planning 
to Action 

With the questions and related data identified, 
you are now ready to specify what will be done, 
by whom, and when. This step involves defining 
meaningful tasks and realistic time lines. An 
important issue is deciding when to collect baseline 
data, which are needed to answer program impact 
and outcome questions. In some cases (e.g., in the 
case of student test scores), data can be collected 
any time during the project. In other cases (e.g., 
teacher attitudes), data must be collected when 
the project begins in order to provide a reference 
point for later comparisons. Also review data 
reports that have already been completed in order 
to make use of existing data in your study. 

Check with your state coordinator to discuss state 
plans for evaluation. Again, data may be available 
through the work of a statewide evaluation and you 
may be able to add questions to an already planned 
survey or interview. If these arrangements can be 
made, you will save considerable time not only 
for the data collectors but also, in some cases, for 
teachers, students, and parents — who are the sources 
of the information. Identify logistical issues that 
must be resolved (e.g., access to records, classrooms, 
and teachers) and specify areas where an outside 
consultant could be helpful. All of the external 
requirements (listed on Worksheet 1) for evaluating 
the reform must be addressed in this design stage. 

Using Worksheet 6 to Create, a, Schedule, 
and KesgorsibiUties jor EimluatioK > Tasks 

Exhibit 6 offers examples of how to use 

Worksheet 6: Task and Responsibility Schedule 

to organize evaluation tasks, designate 
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exhibit 6: Example of Completed Worksheet 6 Worksheet 6: Task and Responsibility Schedule 
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responsibilities, and plan a schedule. To generate 
your list of evaluation tasks for the first column 
of Worksheet 6, review the evaluation questions 
and group them around data sources. Write the 
questions in the last column. Next, designate the 
primary task or tasks associated with the evaluation 
questions. Assign task oversight to members of the 
team, according to their connection to the data 
source or their area of expertise. 

By grouping evaluation questions that have 
common aspects, rather than working with them 
one by one, the evaluation — and, consequently, 
data collection — will be more efficient. One 
natural grouping is questions that rely on the 
same information source. For example, if there 
are several questions that can be answered by 
teachers, perhaps a single teacher survey is the 
most appropriate evaluation tool to efficiently 


collect the data needed to answer them. An 
alternative may be to group questions with a 
common focus. For instance, a context question 
about staff commitment may be conceptually 
connected to an implementation question about 
use of new materials. Thus, you may choose to 
collect data on commitment during interviews 
about use of materials. Of course, some questions 
will necessitate the use of more than one data 
collection method. 

In short, the intent of grouping questions is to 
organize and streamline the data-collection process. 
Still, your evaluation plan needs to be flexible, 
and this is one area you will want to refine as the 
evaluation proceeds. Defining and assigning the 
tasks as part of planning initiates the process. The 
next section, Stage 3, provides guidance on selecting 
and implementing data-collection methods. 
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Stage 3 



Conducting the Evaluation 


t is time to get down to business and put the 
results of your planning and designing efforts 
into practice. Conducting the evaluation entails 
the following steps: 

Step I. Select or create data collection 
instruments. 

Step 2. Collect the data. 

Step 3. Prepare the data for analysis. 

Step 4. Analyze the data. 


Step I. Select or Create Data 
Collection Instruments 

Data collection is the most time-intensive part 
of the evaluation. It is important to do all that you 
can to make it efficient. If staff members with the 
appropriate expertise are not available in-house, 
then secure consultants to oversee instrument 
development, data collection, and data analysis. 

Given the time and cost associated with 
developing data-collection instruments, you 
should use available instruments whenever 
possible. The instruments you need will likely 
include surveys, checklists, observation tools, and 
interview protocols. Even if an existing instrument 
does not quite address all the data needs you have, 
it is easier to adapt it than to develop a new one 
from scratch. If you must develop your own 


instrument, be sure to pilot test it to identify and 
correct unclear instructions and confusing items. 
This can be done with a sample of respondents 
from the same or a parallel source (e.g., a few 
teachers from a different grade level than that 
needed for the evaluation). After you make 
necessary changes, the instrument is ready to use. 

Table 1 lists the advantages and disadvantages of 
common data-collection instruments. Concerns to 
keep in mind include protecting the confidentiality 
of respondent identities, minimizing the time 
demands on both respondents and data collectors, 
and training those who will collect data. Interview 
guides and checklists are a fairly straightforward 
way to ensure consistent data collection across 
collectors and periods. However, it is more difficult 
to ensure data consistency with observations (e.g., 
of classroom teaching). The use of observation as a 
data-collection method requires a data-coding 
protocol and training of observers. 

Assuring confidentiality helps respondents feel 
comfortable with being honest because they 
know there will not be retribution for expressing 
less positive views. It also means that they can 
comment on another persons performance 
without fear of damaging their relationship with 
that person. If your design calls for a pre- or 
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table i: Advantages and Disadvantages of Data-Collection Instruments 


Instrument 

Self-administered 

questionnaires 


Advantages 


I nte r vi ewe r-ad m i n i ste red 
questionnaires 


Open-ended interviews 


Inexpensive. Can be quickly 
administered if distributed to 
group. Well suited for simple and 
short questionnaires. 


Interviewer can probe to 
ensure question is understood. 
With good rapport, may 
obtain useful open-ended 
comments, including evidence 
to support response. 

Usually yield richest data, details, 
new insights. Best if in-depth 
information is wanted. 


Disadvantages 

No control for misunderstood 
questions, missing data, or 
untruthful responses. Not 
suited for exploration of 
complex issues. 

Confidentiality is an issue. 

May require hiring interviewers. 
Training is needed to establish 
consistency, nature, and use of 
probing questions. 


Same as interviewer adminis- 
tered questionnaires. Often 
difficult to analyze. 


Focus groups 

Useful for gathering ideas 
and different viewpoints, 
discovering new insights, and 
improving questions design. 

Not suited for generalizations 
about population being studied. 

Tests 

Provide hard data, which 

Available instruments may be 


administrators and funding 

unsuitable. Developing and 


agencies prefer. Relatively 

validating new project-specific 


easy to administer. Good 

tests may be expensive and time 


instruments may be available 

consuming. Objections may be 


from vendors. 

raised because of test unfairness 



and bias. 

Observations 

If well executed, best for 

Usually expensive. Needs 


obtaining data about behavior of 

well-qualified staff. Observation 


individuals and groups. 

may affect behavior being 
studied. 


Documents, records, 
and student work 


Existing materials can be 
used to develop data at a 
convenient time. 


Checklists or rubrics for 
generating data from written 
material must be created. Careful 
definitions must be established 
±o_ens.u£e_j 


consistency 

Note: Adapted from User-Friendly Handbook for Project Evaluation (p. 44), by F. Stevens, F. Lawrenz, and L. Sharp (n.d, 
Directorate for Education and Human Resources, National Science Foundation. 
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post-survey or test and you wish to maintain 
confidentiality assign ID numbers to the 
respondents and keep a confidential file linking 
names with numbers for use at the time of the 
post-test. Confidentiality makes handling the 
data more cumbersome, given the procedures 
necessary to protect identity. Anonymity is easier 
to assure — there is no identifying name or 
information on the instrument — but then you 
definitely cannot report the same kinds of 
information. And you cannot easily remind 
those who have not yet completed the instrument 
to respond. 

After you have selected or designed your 
instruments, it is important to reexamine them in 
view of the evaluation questions being addressed. 
Too often at this stage, the process and procedures 
of data collection take precedence over the 
substance of the questions to be answered. Be 
sure that all survey and interview items and 
observation protocols focus on the measures 
you have selected to answer each question. In 


addition, because you likely will be relying on 
existing databases or records (e.g., measures 
of student academic performance or student 
attendance), you will want to develop a process 
for selecting the information you need and, in 
some cases, transferring it to a new database 
for use in the evaluation. 

Step 2. Collect the Data 

Data collection involves scheduling, initial 
and follow-up information gathering, and data 
management. Timeliness in completing major 
data-collection and analysis tasks — and reporting 
the results to key stakeholders — is most 
important. Specify when data will be collected, 
from whom, and the steps needed to complete 
the process in your data-collection plan. 

Using Worksheet 7 to Plan Dates Collection 

With Worksheet 6, you prepared a broad 
overview of the evaluation tasks. To ensure 


Tips for Conducting the Evaluation 

■ Create a list or database of contact information for various groups of individuals who will serve 
as sources of data to make ongoing data-collection more efficient. 

■ Carefully plan data analysis procedures at the outset. Consider using a consultant to start on 
the right foot if this expertise is not available in-house. 

■ Document data collection and data analysis procedures early in the evaluation process. You will 
need this information for reporting and for making revisions to the evaluation plan. 

■ Schedule regular team meetings to check on progress with evaluation tasks and the need for 
revision of timelines or for extra support for a task. 

■ Set up a schedule for reporting both for interim reports and final reports. Determine deadlines 
for draft reports and back up from there to assure analyses are completed in time. Also allow 
for quality assurance review of the draft before the team makes it available to others. 
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consistent data collection across all sites and data 
collectors, you can develop precise protocols, 
checklists, logs, and standardized forms using 
Worksheet 7: Data Collection Plan (see 
Exhibit 7). In addition, districts and schools may 
consider providing appropriate staff development 
opportunities to build staff members’ skills in 
data collection and analysis. 

Step 3. Prepare the Data 
for Analysis 

Data come in multiple forms. If you have 
designed your instruments well, you will have 
simplified the preparation steps. Quantitative 
data — from questionnaires, checklists, or extracted 
from records — are generally ready to be entered 
into your database and used in analysis. 

Some checking for data-coding problems is 
needed. Determine and consistently apply strategies 
for handling missing data and obviously incorrect 
data (e.g., a child’s age reported as 30). In addition, 
convert survey questions that require a brief 
narrative response into a form suitable for analysis. 
For example, perhaps you asked teachers to list their 
most frequently used teaching methods. In order to 
reduce the data to meaningful categories, you will 


need to compile a list of responses and then look for 
repeated methods. Assign each category a unique 
identifier (number or letter) so that you will be able 
to report the frequency of each method. The person 
who creates the list must be able to determine when 
different wording indicates the same method. 

Qualitative data require a more labor-intensive 
preparation and analysis process. You may want to 
select software to help prepare and analyze data (for 
a discussion of qualitative analysis programs, see 
Fielding & Fee, 1998). For example, preparation of 
qualitative data is necessary if you conduct parent 
interviews and include open-ended questions about 
parents’ confidence in the school. The responses 
will likely vary in length and substance. Preparation 
of the data entails transcribing interviewer notes 
into a consistent format and then establishing a 
set of categories into which the responses can be 
organized and coded. Responses, for example, 
might be coded on a scale from “no confidence” 
to “extremely confident” that the school is 
providing a quality education. 

Step 4. Analyze the Data 

Too often, schools and districts gather evaluation 


More Tips for Conducting the Evaluation 

■ Don’t rely solely on statistical analysis; in some cases, qualitative analysis may better serve 
your needs. 

■ Allow sufficient time to conduct thoughtful and in-depth analyses and interpretation. 

■ As a team, ask, “Do the results make sense? What are the possible explanations of findings 
and how will the results help us decide what actions to take?” 

■ Involve others in interpreting the results in order to gain insights from their experiences and 
maintain their interest and involvement in the evaluation. 
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EXHIBIT 7: Example of Completed Worksheet 7 (for the year 2010) Worfcslie&t 7: Data Collection Plan 
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be analyzed and the results interpreted. If you have 
kept your design reasonably simple, a first step is to 
report descriptive information. In the case of 
quantitative data, descriptive information typically 
includes frequencies of response (e.g., how many 
selected “strongly agree,” “agree,” “disagree,” 
“strongly disagree,” or “neutral”) and the average, 
or mean, response value. The results of your 
analysis could be reported for all respondents or 
reported for selected subgroups of respondents 
(e.g., teachers, administrators, parents). More 
complex statistical analyses can be used to compare 
groups or subgroups or to note how people who 
responded to one set of questions responded to 
another, related set of questions. 

At its simplest level, analysis is the process 
of combining data from individuals into a 
common result. At another level, analysis 
involves interpreting sets of findings and looking 
across the data to better understand them and 
create a more comprehensive answer to an 
evaluation question. Interpreting data entails 
arriving at a reasonable explanation for the 
results obtained. To arrive at the best and most 
plausible interpretation of findings, consider 
rival explanations and then systematically rule 
out invalid options. 

To illustrate, if survey results reveal that parents 
are not comfortable with the reform process, a 
reasonable conclusion might be that change often 
is difficult and not supported by all stakeholders. 


Another explanation might be that only parents 
who were unhappy with the nature of the reform 
chose to respond to the survey questions. This 
explanation can be checked by examining the 
process used for selecting parent respondents. If 
a variety of parent responses were obtained, the 
responses better represent the views of all parents. 
If, in addition to parent survey responses about 
the reform, the findings from a focus group 
indicate that parents did not like to see things 
change, the conclusion that change is difficult 
for parents is supported. 

The coded qualitative data also provide 
descriptive findings. You will be able to report 
the major themes that result from the data 
collection and whether groups differ in how they 
respond. Qualitative data can often make sense 
of discrepancies found in the quantitative data 
and be used to validate or invalidate alternative 
explanations. The analysis and interpretation 
tasks are among the most important and most 
challenging of the evaluation process itself. 

The team works together to develop explanations 
for the results and to validate them across all the 
data sources relevant to the topic. By presenting a 
draft of interpretations to key stakeholders, such 
as teachers or administrators, you may incorporate 
their comments and suggestions for a more 
complete report. This analysis and interpretation 
process frequently engages stakeholders in 
developing recommendations for improvement. 
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Stage 4 



Reporting the Findings 


N ow that the hard work is done, the 
really hard work begins. It is time to 
let stakeholders know just what you 
are learning about the program, its 
outcomes, and impact. Effective reporting of 
evaluation findings includes the following steps: 

Step I. Organize the study findings. 

Step 2. Select the reporting media and formats. 
Step 3. Frame the findings as 
recommendations. 

Step 4. Deliver/disseminate the findings. 

Step I. Organize the 
Study Findings 

Audiences for the evaluation findings may 
include district administrators and board members; 
parents and community members; and state, 
district, and school staff. With Worksheets 1 and 2, 
you described the data needs and expectations of 
these various stakeholder groups. Use this 
information to determine the types of reports 
you need to develop to best convey your findings 
and recommendations. Most audiences prefer a 
fairly succinct report; some may be interested in 
only one aspect of the information. 

Create a master data file or report that compiles 
all background information, evaluation questions, 


analysis, results, findings, and recommendations. 
This master report provides a foundation for 
meeting various tailored reporting needs. You 
might also keep evaluation reports completed by 
the state department of education or by other 
schools in your district. 

Consider preparing shorter, successive 
evaluation reports that document the 
progress of implementation and immediate 
or interim outcomes. Providing feedback to 
administrators and staff in a timely manner 
will help them make mid-course corrections 
in program implementation. 

Step 2. Select the Reporting 
Media and Formats 

It is important to ensure that evaluation results 
are formatted in a way that stakeholders can readily 
access and use. Develop reports to connect findings 
to particular issues and concerns of government 
agencies, school boards, parents, and other 
stakeholders. For example, principals might prefer 
to receive computer-generated summaries of 
assessments, disaggregated by student groups 
receiving different types of instruction. School 
boards or state officials, on the other hand, might 
prefer statistical progress reports with charts and 
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graphs that compare student performance across 
years or buildings. Parents might prefer a narrative 
report that is formatted to respond to the key 
concerns they have already identified. In addition, 
be sure to develop reports in the languages that are 
relevant to your school community. 

Step 3. Frame the Findings 
as Recommendations 

The evaluation is an integral part of the reform 
process; it should be used to make program 
adjustments and improvements. One of the most 
important ways this can be realized is to reframe 
the findings as specific recommendations for 
program improvement. If you engage stakeholders 
in the interpretation process, recommendations 
may emerge from that discussion. Suggestions for 
modification or change, then, will not come as a 
surprise to these stakeholders. 

Nevertheless, putting recommendations in 
writing will give them more weight and visibility 
with decision makers who have not been as directly 
involved in the evaluation process. For example, if 
there are aspects of the reform effort that are not 


working well, including the recommendations for 
change rather than merely presenting a set of 
findings in the evaluation report will encourage 
participants to make the changes. Developing 
recommendations is neither an easy task nor one 
without consequence for the evaluation team — 
especially when there is substantial resistance to 
change. Working through the implications of the 
recommendations with key decision makers before 
the report is released is important. 

Step 4. Deliver/Disseminate 
the Findings 

If you expect individuals or groups to take 
action, simply providing a written report is seldom 
sufficient. When your findings are important for 
one or more groups to hear, consider presenting 
those findings orally and engaging each group in a 
forum or question-and-answer session. Seek out 
occasions (perhaps as part of previously scheduled 
meetings) to keep your audiences informed. Placing 
articles in newsletters and posting information on 
available websites are effective strategies for keeping 
stakeholders informed and interested. 


Tips for Reporting Evaluation Findings 

■ Create a master report or data file that compiles all the local context, program characteristics, 
baseline data, and evaluation findings. Such a file is relatively easy to update with new data and 
findings as necessary. 

■ Use the master report to create reports and presentations tailored to key audiences. Consider 
developing (at least) 

• a written executive summary of the findings, 

• an oral presentation supported with overheads or a computer-based presentation, and 

• a briefing paper that highlights key evaluation findings and recommendations. 

■ Don’t forget to include recommendations for program adjustments and improvements in your 
reports and presentations. They are the blueprint for the future. 
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Stage 5 


Encouraging Use 
of the Findings 


A t this point, you’ve completed all Step I. Create Opportunities to 
the steps for the first four stages DiSCUSS Findings 
and have reported the findings. Create opportunities to discuss evaluation 

You are in the home stretch, but findings and their use with stakeholder groups, 

you’re not quite done. That’s because reporting Be sure to ask that sufficient time be made 

the findings does not ensure that they will be available to present findings and to discuss 

used in any meaningful way. The value of the recommendations. Keep your presentation short, 

work you have accomplished so far rests in but be specific about suggested changes. Allow 

improving the program each time the evaluation ample time for discussion, especially with the 

indicates an adjustment is needed. Encouraging teaching staff. You may even choose to provide 

stakeholders to use the findings means enabling a draft of the report to the group and offer to 
them to sharply focus their practices for an even incorporate feedback in the final version. In 
better result. In order to encourage stakeholders addition, you might enlist the help of students 

to put the evaluation to use, consider taking in reporting findings to parents, especially if they 

the following steps: have had a role in data collection. 

Step I. Create opportunities to 

discuss findings. Using Worksheet 8 to StmwiArvce Actions 

Step 2. Follow up with stakeholders to 

determine how findings are used. Exhibit 8 illustrates how to use Worksheet 8: 

Step 3. Revise the plan for ongoing program Summary of Actions to Encourage Use of 

evaluation as necessary. Findings to plan and track your efforts. Building 

a broad base of support for the recommendations 
helps everyone, including administrators, take 
action. Therefore, be sure to plan to present the 
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recommendations to a variety of audiences and 
key individuals. 

Step 2. Follow Up with 
Stakeholders to Determine 
How Findings Are Used 

Follow up with key members of stakeholder 
groups to determine how they are using the 
findings and what other information needs they 
may have. On Worksheet 8, record the actions 
your team takes to encourage application of 
the findings. This is an essential step because, 
if evaluation findings are not used in the early 
stages of the program, the reform may not be 
fully implemented. For example, if teachers lack 
needed materials and no attempt is made to 


supply them, they cannot possibly implement 
the program as planned. In later stages of program 
implementation, more outcome results become 
available, so you will know even better what 
works and what does not work. Incorporate 
these findings and recommendations into your 
long-range school-planning efforts. 

Step 3. Revise the Plan for 
Ongoing Evaluation as Neces- 
sary 

Once you have determined how the findings 
were used and what other information might 
be helpful to stakeholders, revise the evaluation 
plan. Of course, the evaluation plan also needs 
to be revised whenever the reform program itself 
is revised. 


Tips for Encouraging the Use of Findings 

■ Be sure that findings and recommendations are tailored to be used by specific 
stakeholder groups. 

■ Be specific and give examples of what might be done in the future in your presentations 
and reports. 

■ Let each stakeholder group know that using the findings is important and that you will be 
following up with them to determine if you can be of assistance. 

■ Always talk with key members of stakeholder groups to gain new insights about what is 
important to them and what might be added to the ongoing program evaluation. 
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exhibit 8: Example of Completed Worksheet 8 Worksheet 8: Summary of Actions to Encourage Use of Findings 
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Worksheet 1: Alignment of Evaluation Requirements 
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Worksheet 2: Evaluation Planning Questions 
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2010 McREL 



Worksheet 3: Comprehensive School Reform Program Logic Model 
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Worksheet 4: Summary of Evaluation Questions 
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Worksheet 5: Evaluation Design Matrix 
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Worksheet 6: Task and Responsibility Schedule 
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Worksheet 7: Data Collection Plan 



Sustaining Coherent Reform An Evaluation Guide for Districts and Schools 


51 


2010 McREL 




Worksheet 8: Summary of Actions to Encourage Use of Findings 



Sustaining Coherent Reform An Evaluation Guide for Districts and Schools 


53 


2010 McREL 















Appendix B 







Reporting 

l and 1 

Evaluation 



Requirements f 

or ! 

School 



Improvement Grants 

i 
































































































































Sustaining 

Coherent 

Reform 

An Evaluation 
Guide for 
Districts and 
Schools 


55 


APPENDIX B: Reporting and Evaluation Requirements for School Improvement Grants 


Achievement 

Indicators 


Leading 

Indicators 


SCHOOL DATA 


Which intervention the school used (i.e., turnaround, restart, 
closure, or transformation) 


NEW SIG 


AYP status 


ED Facts 


/ 


Which AYP targets the school met and missed 


ED Facts 


/ 


School improvement status 


ED Facts 


/ 


/ 


Number of minutes within the school year 

r 


NEW SIG 




STUDENT 0 U T C 0 M E / A C A D E M I C PROGRESS 


Percentage of students at or above each proficiency level on state 
assessments in reading/language arts and mathematics (e.g., Basic, 
Proficient, Advanced), by grade and by student subgroup 


ED Facts 


/ 


/ 


Student participation rate on state assessments in reading/language 
arts and in mathematics, by student subgroup 


ED Facts 


/ 


Average scale scores on state assessments in reading/language arts 
and in mathematics, by grade, for the “all students” group, for each 
achievement quartile, and for each subgroup 


NEW SIG 


/ 


Percentage of limited-English-proficient students who attain English 
language proficiency 


ED Facts 


/ 


Graduation rate 


ED Facts 


/ 


Dropout rate 


ED Facts 


/ 


Student attendance rate 


ED Facts 


/ 


Number and percentage of students completing advanced 
coursework (e.g.,AP/IB), early-college high schools, or dual 
enrollment classes 


NEW SIG 
HS only 


/ 


College enrollment rates 


NEW SFSF 
Phase II 
HS only 


/ 
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Metric 

Source 

Achievement 

Indicators 

Leading 

Indicators 

STUDENT CONNECTION 

AND SC 

H 0 0 L CLIMATE 

Discipline incidents 

ED Facts 


/ 

Truants 

ED Facts 


/ 



TALENT 

Distribution of teachers by performance level on LEA’s teacher 
evaluation system 

NEW SFSF 
Phase II 


/ 

Teacher attendance rate 

NEW SIG 


/ 
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Resources I J }. 

Books and Articles 

Augustine, C. H., Gonzalez, G., Ikemoto, G., Russell, J., Zellman, G., Constant, L., Armstrong, J., 

& Dembosky, J. W. (2009). Improving school leadership: The promise of cohesive leadership systems. 
Retrieved from www.wallacefoundation.org 

Center on Innovation and Improvement. (2007). 

School turnarounds: A review of cross-sector evidence on dramatic organizational improvement. 

Lincoln, IL: Author. 

Coggshall, J., Lasagna, M., & Laine, S. (2009). Toward the structural transformation of schools: 

Innovations in staffing. Chicago, IL: Learning Point Associates. 

Herman, J. L., & Winters, L. (1992). Tracking your schools success. Newbury Park, CA: Corwin Press. 

Murphy, J. (2009). Turning around failing school: Policy insights from the corporate, government, 
and nonprofit sectors. Educational Policy, 23(6), 796—825. 

Sanders, J. R. (2000). Evaluating school programs: An educator s guide (2nd ed.). Newbury Park, CA: 
Corwin Press. 

Stevens, E, Lawrenz, E, & Sharp, L. (n.d.). User-friendly handbook for project evaluation: Science , 
mathematics , engineering and technology education. Washington, DC: Directorate for Education 
and Human Resources, National Science Foundation. 

Tesch, R. (1990). Qualitative research: Analysis types and software tools. Oxon, England: RoutledgeFalmer. 

W. K. Kellogg Foundation. (1998). Evaluation handbook. Battle Creek, MI: Collateral 
Management Company. 

Websites 

Assessment and Accountability Comprehensive Center (AACC) 

http://www.aacompcenter.org/ cs/ aacc/ print/htdocs/ aacc/home.htm 

The AACC implements, evaluates, and improves assessment and accountability systems so that 
states and districts can reach the goal of academic proficiency for all students. 

Center for Data-driven Reform in Education (CDDRE) 

http://www.cddre.org 

CDDRE conducts research into how school districts can use data-driven reform to improve 
student achievement. 

Center on Innovation and School Improvement 

http://www.centerii.org 

This site provides access to many resources and serves as a clearinghouse for state school improvement 
teams and efforts. 

Center on Instruction 

http : / / www. centeroninstr uction . org 

This site serves as a gateway to a collection of scientifically based research and information on K-12 
instruction in reading, mathematics, science, special education, and English-language learning. 

Resources continued 
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McREL 

http://www.mcrel.org 

This site provides information related to the wide range of products and services provided by McREL, 
a Denver-based private, nonprofit corporation dedicated to making a difference in public education. 

National Center for Research on Evaluation, Standards, and Student Testing (CRESST) 

http : / / www. cse . ucla. edu 

This site provides information about the development, validation and use of sound data for improved 
accountability and decision making and explores technological applications to improve assessment and 
evaluation practice. 

National Comprehensive Center for Teacher Quality (TQ Center) 

http://www.tqsource.org/ 

This site has resources related to improving the quality of teaching, especially in high-poverty, 
low-performing, and hard-to-staff schools. 

National High School Center at the American Institutes for Research 

http://www.betterhighschools.org/ default.asp 

This site is devoted to research and best practices for high schools and serves the regional 
comprehensive centers. 

Regional Education Laboratory Program 

http://ies.ed.gov/ ncee/ edlabs 

This site provides an overview of and access to the information and resources available to states within 
each of the ten regional education laboratories. 

U. S. Department of Education 

http : / / www. ed. gov 

School Improvement Fund 

http://www2.ed.gov/ programs/ sif/index.html 

This page provides access to the necessary information and resources related to School Improvement 
Fund formula grants. 

Helpful ED and ED-Junded Publications and Resources 

http://www2.ed.gov/ policy/ gen/leg/ recovery/ publications.html 

This page provides access to resources related specifically to the American Reinvestment and Recovery 
Act and the related funding and school reform efforts. 

Comprehensive School Reform Program 
http://www2.ed.gov/ programs/ compreform/index.html 

This page describes the CSR program and types of projects it entails. The CSR program is autho- 
rized to provide formula grants to SEAs to award competitive grants to LEAs on behalf of schools. 
However, in FY 2008, funding was appropriated only for the CSR Clearinghouse, which provides 
support for comprehensive school reform activities. 
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